Skip to content

feat: add scheduler backend - #1282

Merged
kvlonge merged 225 commits into
mainfrom
feat/orchestration-backend
Nov 5, 2026
Merged

kvlonge merged 225 commits into
mainfrom
feat/orchestration-backend

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Why

Native orchestration: a fenced scheduler, Python workers and an authenticated control API, now folded with project roles, changed-only builds and non-Linux recovery so one branch merges to main.

Changes

  • SQLite/PostgreSQL control store, fenced scheduling, immutable snapshots, API/watchdog, audited administration.
  • Pinned workers; atomic groups, frozen retries/backfills, calendars, webhooks, sensors.
  • Retention for audit rows, ticks and snapshots; whole-second cron ticks; one latest catch-up tick.
  • Roles: password/OIDC/proxy sign-in, custom roles with approvals, project roles from roles.toml with node selectors, OIDC group roles (beta).
  • sqb build --changed and the job changed setting, decided from append-only _sqlbuild_consumption records of each write's inputs.
  • sqlbuild-process-ownership: worker ownership, recovery and stop on Linux, macOS and Windows; Verify job "Scheduler recovery (macOS, Windows)".
  • Scheduler contracts renew leases held across slow setup; the 30-second lease is unchanged.
  • Single-use browser-approved CLI sessions, OS credential storage.
  • Not included: signed approvals; operator backfill past tick retention.

Verification

  • Final head, locally: make check-ci; make test 12,009 pass, 2 skipped (six signal tests run with default SIGINT handling); make test-native for state and orchestration; hygiene scan.
  • Simplification pass (no behaviour change): consumption rows dedupe on their natural key instead of a row event ID; shared relation and ranked-execution helpers. Adapter consumption DDL overrides stay, as the strict adapter contract requires.
  • Fenced-lease fix: a forced 35 s stall fails Fenced without lease renewal and passes with it.
  • Native orchestration contracts on SQLite and PostgreSQL, including scheduler faults, DST, latency and upgrade.
  • Hosted Verify on this head covers Windows and macOS recovery.

Preserve in-progress storage, scheduling, claims, credentials and native API/CLI work. The orchestration backend remains incomplete and is not ready for delivery.
Add claim-scoped execution inputs and native snapshot drift validation.

Incomplete orchestration backend checkpoint; worker launch and the remaining control-plane integration are not implemented yet. Targeted SQLite/PostgreSQL tests pass; full static readiness is pending.
Verify pinned inputs before start and expose retryable run reports and cancellation heartbeats over the authenticated API. Disable redirects for credentialed requests and cover the lifecycle against SQLite and PostgreSQL. Keep the orchestration PR explicitly incomplete pending actual launchers and the remaining backend scope.
Separate lease expiry, replica connections, terminal run fixtures and fault cases so each integration binary imports only the support it uses. Preserve SQLite and PostgreSQL transaction coverage.
Observe heartbeat loss in fenced, bounded transactions while retaining physical write claims. Recover through existing worker heartbeats and continue full batches without an idle delay.
Own worker heartbeats on a dedicated native runtime and connection pool. Propagate cancellation and revoked credentials without cancelling work on transient API failures; stop and join with the execution owner.
Move bounded client construction, credential redaction, URL validation and JSON response handling to the API owner so worker and location flows can share them. Keep independent heartbeat pools and worker-facing error categories.
Add privileged native administration with JSON results, bounded location credentials and atomic security audit. Detect lost result delivery without claiming committed mutations rolled back.
Keep the public Python API limited to authoring records and native delegation. Resolve selectors and validate schedule data in Rust before publication. Preserve stable shorthand trigger names across cron edits.

Add the explicitly approved narrow Python facade boundary entries without increasing movable Python or native crossing budgets.
Collect exported jobs and nested factory containers natively, deduplicating aliases and terminating cyclic containers. Explicit empty declarations skip user module execution. Remove failed imports from the module cache so publication retries cannot reuse partial definitions.
Verify retained compiler baselines before worker startup and join native heartbeat work on interpreter-side close. Preserve claim identity and report retry keys without treating resource teardown as physical warehouse completion.
Restrict forward and reverse cron expansion and pure tick planning to the scheduler. Preserve scheduling bodies and all existing fault, DST and simulation coverage.
Expose owner-only cancellation with stable retry identity. Commit lifecycle facts, projections and security audit together while retaining active physical claims until termination.
Resolve manual requests against the current published job inside one write transaction. Preserve the accepted snapshot and execution inputs across retries after redeployment, and commit queue publication with the trusted owner audit.
Compile in the project interpreter and retain native snapshots through publication. Keep JSON results separate from progress and distinguish uncertain acknowledgements from failed result delivery.
The decision rules were a trait with one real implementation and four
broken ones that only the simulation test used. The shipped rules are
now plain code. The simulation still proves its oracle finds misses
by writing consumption without the started row.
Appends now go in fixed-size statements: keeping an execution inside
one statement bought nothing, because separate statements are not
atomic together and the decision already treats a partly written
execution as unfinished. Source freshness and consumption share one
state-table naming helper, and seeds use the same forced-run reasons
as models. The consumption connection's close joins the allowlist of
context objects that each release their own resource.
…e place

Each SQL Server state index repeated the sys.indexes guard, and each
keyed state table repeated the NVARCHAR(450) key narrowing. Postgres
repeated CREATE INDEX IF NOT EXISTS per table. One private builder per
adapter now takes the table, index and columns, and the fingerprint,
freshness, node-result, audit-result and consumption methods use it.
The rendered SQL is unchanged.
The process-ownership crate follows the workspace's fixed 0.0.0 crate
version, like every other unreleased crate.
The Windows recovery job failed six orchestration tests with Fenced.
Run serially on the same runner, five of them passed: four store-heavy
tests at once on the four-core runner starved each other past the
30-second scheduler lease. The job now runs two at a time there.

The sixth failed serially as well. The tick-retention test inserts a
backlog of a thousand ticks one at a time after taking the lease, and
on that runner the inserts outlasted it. The scheduler renews its lease
between passes, and the test now renews before pruning in the same way.

The temporary latency and serial-timing diagnostics are removed, and so
is the CLI step whose filter no longer matched any test.
An OIDC server can now map identity-provider groups to roles.
groups_claim names the ID-token claim that lists a person's groups,
and each group_roles entry maps one group to a built-in or custom
role, optionally limited to one location and target. A mapping cannot
grant admin, and a custom role or a target needs a location.

Groups are read only from the verified ID token, never from userinfo.
A claim that is not a list of nonempty strings, including an overage
pointer, gives no groups. People still need an explicit identity
binding; groups never create accounts.

Each sign-in replaces the person's recorded groups, and the server
publishes its mapping at startup, so authorization joins the two live:
removing a mapping narrows access at once, and leaving a group takes
effect at the next sign-in. Other sign-in methods publish no mapping.
# Conflicts:
#	crates/sqlbuild-orchestration/src/authorization/main/authorize_person.rs
Explain groups_claim and group_roles, which roles and scopes a mapping
may grant, and when changes take effect: groups are replaced from the
ID token at every sign-in, and a mapping change applies when the server
restarts. Servers sharing one control database must use identical OIDC
and group-role configuration.
ISO strings with and without fractional seconds do not sort as instants,
so parse both sides before comparing.
Store transactions on the Windows runner ran about 20 times slower than on
Linux, so tests holding one scheduler lease across their setup were fenced
regardless of test thread count.
Four contracts hold one 30-second lease across many store transactions.
On a slow runner the lease expired before the call under test and the
contract failed Fenced. A background renewal, as a running scheduler
does, keeps the epoch; LEASE_TTL_SECONDS is unchanged.
Turning it off made the store contracts faster but did not stop leases
expiring; the contracts now renew their leases instead.
# Conflicts:
#	tests/integration/src/sqlbuild/compiler/compiled_project/publication/_test_types.py
#	tests/integration/src/sqlbuild/compiler/compiled_project/publication/helpers.py
@kvlonge kvlonge changed the title feat: add scheduler backend (incomplete) feat: add scheduler backend Nov 5, 2026
Deduplicate retried rows on execution, phase and input instead of a
stored event ID; the consumption table has not shipped. The bindings
reuse the project's node relations and one set of field readers, and
the recent and prune queries share their ranking subquery.
@kvlonge
kvlonge marked this pull request as ready for review November 5, 2026 21:46
@kvlonge
kvlonge enabled auto-merge (squash) November 5, 2026 21:46
@kvlonge
kvlonge merged commit 5f451f8 into main Nov 5, 2026
50 checks passed
@kvlonge
kvlonge deleted the feat/orchestration-backend branch November 5, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant